Hello!
Welcome to my first internal training. In this markdown, we will learn about Social Network Analysis using tidygraph (including igraph and ggraph). We’ll not only learn about the visualizing stuff but also the metrics. We’ll analyze Twitter network as our study case using rtweet package. After this internal training, I hope we will be able to do:

  • Analyze Ego Network from specific account/user
  • Analyze Information Activity Network from something that goes viral

Note: On 28 March 2019, Kak Eca already deliver a cool internal training about Twitter Interactions using Twinetverse. It has a similar idea to this study, we both explain about building networks from Twitter data. The difference in this study is we do not only make visualization but also calculate metrics to achieve the goals for every case (spoiler: we’ll do 2 SNA cases). We’ll also more focus on how to implement SNA and what it is. This study is theory-driven studies.

1 Background

1.1 Introduction

What is Social Network Analysis (SNA)
A social network is a structure composed of a set of actors, some of which are connected by a set of one or more relations. Social network analysis work at describing underlying patterns of social structure, explaining the impact of such patterns in behavior and attitudes. Social network analysis has 4 main types of network metrics, namely:

  • Network Models: Describe how to model the relationship between users
  • Key Players: To identify the most influential users in the network based on a different context
  • Tie Strength: To measure the strength of a user’s relationship
  • Network Cohesion: To measure how cohesive entities in the networks towards network behavior.

So what? Why do we need them?
Humans are social beings. Even when you sleep, you’re still connected to everyone in the world by your smartphone. Your smartphone keeps sending and receive information like weather information, incoming whatsapp messages, late-night One Piece update, and social media notification from your favorite bias. We’re always connected and there’s network everywhere. Somehow, some smart dudes behind the famous Small World Theory found something from the network that quite exciting.

Did you know you only seperates by six steps from your favorite person in the world? We are able to quantify the what-so-called network and can be implemented in many fields. In this study, we’ll only focus on identify network metrics with key player as the expected output (see the 4 main types of network metrics above). Here’s some implementation of SNA to enlight your knowledge about SNA a bit:

2 Let’s Begin !

2.1 Prerequisites

We’ll crawl Twitter data using Twitter’s rest API. Thus, we need authentication to use the API. To access the API, you will need to create a Twitter Developer Account here: https://developer.twitter.com/en (make sure you already have Twitter account). Creating a Twitter developer account is simple and tends to be fast but it depend on how you describe what you will do with the API.

Good news! recent update of rtweet allows you to interact with Twitter API without creatin your own Twitter developer account. But it’s better if you have one because it gives you more stability and permissions. If you need further explanation, you can head over rtweet’s official website here.

In this study, we’ll crawl Twitter data without using access token as credentials. but if things are going bad, I provide some access token we can use to crawl the data. Note: Due to the rate limit, and most of us will use it at the same time, please use the token wisely.

## <Token>
## <oauth_endpoint>
##  request:   https://api.twitter.com/oauth/request_token
##  authorize: https://api.twitter.com/oauth/authenticate
##  access:    https://api.twitter.com/oauth/access_token
## <oauth_app> Automated Twitter SNA
##   key:    A5csjkdrS24vJ5ktiKYtgasFY
##   secret: <hidden>
## <credentials> oauth_token, oauth_token_secret
## ---

2.2 Graph Theory

In mathematics, graph theory is the study of graphs, which are mathematical structures used to model pairwise relations between objects. A graph in this context is made up of vertices (also called nodes or points) which are connected by edges (also called links or lines). In general, A connection between nodes are separated by 2 types: Directed and Undirected.

  • Directed is a relationship between nodes that the edges have a direction (The edges have orientations). You will recognize it as edges that have an arrow in it. Directed network also separated into 2 types based on its direction, namely: in-degree and out-degree. In-degree represents the number of edges incoming to a vertex/node. In below directed graph, In-degree of A is 1 and degree of D is 2. Out-degree represents the number of edges outgoing from a vertex. In below directed graph, out-degree of A is 1 and out-degree of C is 3.

  • Undirected indicates a two-way relationship, the edges are unidirectional, with no direction associated with them. Hence, the graph can be traversed in either direction. The absence of an arrow tells us that the graph is undirected.

3 Case 1: TeamAlgoritma Ego Network

Ego network is a concept indicates the amount of all the nodes to which an ego/node is directly connected and includes all of the ties among nodes in a network. You take any random username/company/person you want to analyze, gather all their neighborhood, and analyze it. sometimes you’ll find interesting patterns like this person has a lot of different communities and none of them are look-alike, or you can also found a person who can spread information most widely around your target-person network.

Case objectives:
- Analyze Ego Network from @TeamAlgoritma Twitter account
+ Visualize top cluster from TeamAlgoritma mutual account
+ find out which account has the potential to spread information widely
+ Calculate the metrics, and find out who is the key player in TeamAlgoritma network

Here’s the step to do this case:
1. Gather TeamAlgoritma detail Twitter data
2. Gather all TeamAlgoritma followers
3. From the follower, filter to active account only and gather their follower and following
4. Create Mutual data from following and follower data
5. Build communities, Calculate SNA metrics, and identify which user is important
6. Visualize the ego network

Picture 1: Ego network

Picture 1: Ego network

3.1 Gather @TeamAlgoritma data

3.2 Gather TeamAlgoritma followers

## 75000 followers!

TeamAlgoritma Twitter account has 342 followers (on 15 May 2020). We need to gather all of their follower and following but Twitter rest API has (kinda stingy) limitation.

We can only gather 15 users (both following and follower) and 5k retrieved for every 15 minutes, so you can imagine if we want to retrieve thousand of them.. In order to minimize the time consumption, we need to filter the users to active users only. The criteria of ‘active users’ depend on your data. You need to lookup which kind of users your follower is and build your own criteria. In this case, the top 8 of Algoritma’s followers is a media account. Accounts like ‘btekno’ and ‘machinelearnflx’ only repost link to their own media and never retweet other account tweets. So if our goal is to map the potential information spreading around TeamAlgoritma ego network, we need to exclude them for that reason.

After a long inspection, i propose several criteria for filtering active account: Followers_count > 100 and < 6000, following_count > 75, favourites_count > 10, and create a new tweet at least 2 months ago. I also want to exclude protected accounts because we actually can’t do anything about it, we can’t gather their following and followers.

3.3 Gather TeamAlgoritma follower’s follower

I build a loop function to gather followers from a list. Actually, we can gather the follower with this simple code

But we want to minimize the total user we want to retrieve (n parameter). i build a simple function to retrieve half of the followers if they have more than 1500 followers, and 75% followers if they have less than 1500

We also want to avoid SSL/TLS bug while we gather the followers. Sometimes when you reach the rate limit, the loop tends to crash and stop running. To avoid that, i order the loop to sleep every 5 gathered account (it doesn’t always solve the problem, but it way much better)

After gathering, bind the list to dataframe, convert the username to user_id by left_join from active_fol data, and build clean data frame without NA

The loop need a looong time to be done. To speed up our progress, i already gather the followers and we’ll use it for analysis.

3.4 Gather TeamAlgoritma follower’s following

Same as before, we build a loop function to gather the following. in rtweet package, following is also called as friend.

As you can see, friends_count is way more higher than followers_count. Thus, we need to specify how many users we want to retrieve (n parameter). We want to minimize it, i change flt_n function to gather only 40% if they have more than 2k following, and 65% if less than 2k.

The loop is also a bit different. instead of list, we store the data to dataframe. get_friends() function gives 2 columns as their output; friend list and the query. we can easily just row bind them.

This loop also takes a long time to run. Again, to speed up our progress, we will use the following data i already gathered.

We need to make sure all unique active user in algo_friend is available in algo_following and vice versa

3.5 Create Mutual dataframe

Now we have both following and follower data. We need to build ‘mutual’ data to make sure the network is a strong two-side-connection network. Mutual is my terms of people who follow each other. we can found that by: split algo_friend data by every unique active_user, then we find every account in the following column that also appears in algo_follower$follower. The presence in both column indicates the user is following each other

It isn’t done yet. this is an ego network for TeamAlgoritma account, we want that account to appear on our screen. since TeamAlgoritma barely follows back its followers, it’s not a surprise if we can’t found it in mutual dataframe

So we need to add them manually. we already have un_active dataframe contain unique value of active users. we can simply add extra column contain ’TeamAlgoritma" then bind them with algo_mutual df

phew, we finished the data gathering step! next, we’ll jump into SNA process

3.6 Build nodes, edges, and graph dataframe

A network consists of nodes and edges. nodes (also called vertices) indicates every unique object in network and edges is a relation between nodes (object). We’ll build nodes dataframe from every unique account in algo_mutual df. and edges dataframe that contains pair of accounts, we can use algo_mutual df for that.

after that, we can simply create graph dataframe using graph_from_data_frame function from igraph package.

3.7 Build communities and calculate metrics

I need to remind you we’ll do the analysis using tidygraph style. There are lots of different code styles to build a network but i found tidygraph package is the easiest. tidygraph are just wrappers for igraph packages.

igraph code example:

Create communities using group_walktrap() algorithm, and calculate lots of metrics using tidygraph style

## # A tbl_graph: 14542 nodes and 15975 edges
## #
## # An undirected multigraph with 1 component
## #
## # Node Data: 14,542 x 6 (active)
##   name                community degree_c betweenness_c closeness_c    eigen
##   <chr>               <fct>        <dbl>         <dbl>       <dbl>    <dbl>
## 1 35167068            2               20     0.00252         0.267 0.00206 
## 2 2196972205          2                7     0.000195        0.258 0.000743
## 3 97163422            2                9     0.000458        0.260 0.000960
## 4 1233994338922684416 2               23     0.00554         0.270 0.00254 
## 5 882769518061395968  2                5     0.0000438       0.256 0.000557
## 6 573357589           2                2     0               0.254 0.000263
## # ... with 1.454e+04 more rows
## #
## # Edge Data: 15,975 x 2
##    from    to
##   <int> <int>
## 1     1  6123
## 2     2  6123
## 3     3  6123
## # ... with 1.597e+04 more rows

We can easily convert it to dataframe using as.data.frame() function. We need to this to specify who is the key player in TeamAlgoritma ego network

3.8 Graph Metrics

Before we make a conclusion from the table above, let’s take a time to learn what’s the idea behind those metrics. We’ll build a network from Algoritma Product Team as dummy network to make the explanation easier, and just to inform you how SNA works in real case

3.8.2 Closeness centrality

The closeness centrality of a node is the average length of the shortest path (geodesic) between the node and all other nodes in the graph. Thus the more central a node is, the closer it is to all other nodes. \[C(i) = \frac{N-1}{\sum_{j}d(j,i)}\] \(d(j,i)\) is the distance between vertices \(j\) and \(i\). This centrality divide total number of nodes minus 1(\(N-1\)) by total number of every shorthest path between one node to every node in the graph.

3.8.3 Betweenness centrality

Betweenness centrality quantifies the number of times a node acts as a bridge along the shortest path between two other nodes/groups.
\[C_{B}(v) = \sum_{ij}\frac{\sigma_{ij}(v)}{\sigma_{ij}}\] Where \(\sigma_{ij}\) is total number of shortest paths from node \(i\) to node \(j\) and \(\sigma_{ij}(v)\) is the number of those paths that pass through \(v\)

3.8.4 Eigenvector centrality

Eigenvector centrality is a measure of the influence of a node in a network. The relative score that is assigned to the nodes in the network is based on the concept that connections to high-scoring contributes more to the score of the node in question than equal connections to low-scoring nodes. This amazing link will help you with the calculation.

if \(A\) is the adjency matrix of a graph and \(\lambda\) is the largest eigenvalue of \(A\) and \(x\) is the corresponding eigenvector then \(Ax = \lambda x\). it can be transformed to \(x = \frac{1}{\lambda}Ax\). where \(Ax\) can be defined \(\sum_{j=1}^{N}A_{i,j}x_{j}\) therefore: \[C_{E}(i) = x_{i} = \frac{1}{\lambda}\sum_{j=1}^{N}A_{i,j}x_{j}\]

3.8.5 Community and Modularity

Building community in graph theory is a bit different than clustering in machine learning.igraph package implements a number of community detection methods, community structure detection algorithms try to find dense subgraphs in directed or undirected graphs, by optimizing some criteria and usually using heuristics. Community detection algorithm like group_walktrap(), group_fast_greedy(), and group_louvain() has their own way to create communities in the network. One of the common use community detection algorithm is group_walktrap(). This function tries to find densely connected subgraphs, also called communities in a graph via random walks. The idea is that short random walks tend to stay in the same community.

Modularity in the other hand, is a measure of how good the division is, or how separated are the different vertex types from each other \[Q = \frac{1}{2m}\sum_{ij}(A_{ij}-\frac{k_{i}k_{j}}{2m})\delta(c_{i},c_{j})\] here \(m\) is the number of edges, \(A_{ij}\) is the element of the \(A\) adjacency matrix in row \(i\) and column \(j\), \(k_{i}\) is the degree of \(i\), \(k_{j}\) is the degree of \(j\), \(c_{i}\) is the type (or component) of \(i\), \(c_{j}\) that of \(j\), and \(\delta(c_{i},c_{j})\) is the Kronecker delta, which returns 1 if the operands are equal and 0 otherwise. In summary, networks with high modularity have dense connections between the nodes within community but sparse connections between nodes in different community

Now let’s check if this network has high or low modularity score

## [1] 0.08518007

low modularity score indicates the community in the network actually don’t have much difference. There are dense connections between nodes in both communities. The community member might be different depends on what algorithm you use. you can try different algorithms and compare them using compare() function. By default, compare() returns a score by its Variance Information (method = "vi), which counts whether or not any two vertices are members of the same community. A lower score means that the two community structures are more similar

## [1] 0.6614012

3.9 Identify prominent user in the network

So at this point, i hope you understand the concept of graph, nodes & edges, centrality, community & modularity, and how to use it. We will move back to our Twitter network. We already convert the table_graph to data frame. Last thing we need to do is to find top account in each centrality and pull the key player

Key player is a term for the most influential users in the network based on different contexts. ‘Different context’ in this case is different centrality metrics. Each centrality have different use and interpretation, a user that appears in the top of most centrality will be considered as the key player of the whole network.

From the table above, account “1049333510505291776” appears in most centrality. That account has the most degree in the network (high degree) but also surrounded by important persons (high eigenvector). TeamAlgoritma is an exception, that’s our ego query so it isn’t wise to make it as a key player in its own ego network. We can conclude that user “1049333510505291776” is the key player of TeamAlgoritma Twitter ego network.

Note: after lots of troubleshooting, it seems that user-id “1049333510505291776” has been changed to “1049333510505291778”. I use old data when i do this analysis (15Mmay 2020) and twitter user-id is changed over time. Let’s see who he/she is:

3.10 Visualize Network

Let’s try to visualize the network. I’ll scale the nodes by degree centrality, and color it by community. since our network is too large (approximately 14k nodes and 15k edges), i’ll filter by only showing community 1 - 3.

This process take 30-40 minutes in my pc. i save the plot in rds file to speed up our process

What can we get from this visualization? This obviously doesn’t tell much of a story (we need further inspection in the data, matching it to the visualization), but it shows that the “random walk” community detection algorithm is picking up on the same structure as “stress” layout algorithm. TeamAlgoritma as our ego appears in the middle, act as a bridge who connects all cluster. we only show user label who has high betweenness centrality value. a mushroom-shaped nodes behind them are their mutual friends who don’t follow TeamAlgoritma account. That user is our potential reader if their ‘bridge’ retweeting or mentioning something about TeamAlgoritma account. user in the same community or who close to each other maybe know each other in real life. they create their own community. The key player is in community #1 (red), which is TeamAlgoritma’s most important community because they have the most potential to spread information fast and widely.

4 Case 2: Activity Network

Activity network analyzes how information is spread around the network. From this analysis, we can found how something goes viral and how people interact with it. Because this network contains all forms of Twitter communication (even a single retweet), the visualization will most likely become a hairball. You will need a lot of data featuring or filtering for visualization purposes, but in the end it will be up to your preferences.

4.1 Case Objective

  • Analyze Activity Network given random keyword(s) or hashtag
    • Visualize the activity/information network
    • find out which cluster talk about which topic regarding the keyword
    • Calculate the metrics, and find out who is the key player in the whole conversation network

Here’s the step to do this case:
1. Gather any trending keywords or hashtag (in this case i’ll use #NewNormal)
2. Filter the data, use tweet with lots of interaction (retweet or mention)
3. Specify which column is communicating with which, then create edges df
4. Create nodes df and the network
5. Build communities, Calculate SNA metrics, and identify which user is important
6. Visualize the network

4.2 Gather tweets data

You can always choose other keywords or hashtags. in this example, i’ll use #NewNormal as the query.

it’ll take approximately 4-5 minutes and there’s a possibility of failure (sls/tls, or simply bad connection). so i provide the output in csv for us to continue the analysis.

4.3 Build edges, nodes, and graph dataframe

mentions_screen_name column contains user screen_name we interact with. it’ll be our “to” column in edges. however, by default the column is list type (that i convert to character). We need to remove unused string first. Here’s a function to remove all symbols except letter, number and comma ’ , ’. it also remove first letter in the string (c in our case)

then we simply apply it to the column

## Warning in graph_from_data_frame(d = edge_nn, vertices = nodes_nn, directed =
## T): In `d' `NA' elements were replaced with string "NA"
## Warning in graph_from_data_frame(d = edge_nn, vertices = nodes_nn, directed =
## T): In `vertices[,1]' `NA' elements were replaced with string "NA"
## Warning in closeness(graph = graph, vids = V(graph), mode = mode, weights =
## weights, : At centrality.c:2784 :closeness centrality is not well-defined for
## disconnected graphs
## # A tbl_graph: 17384 nodes and 20304 edges
## #
## # A directed multigraph with 948 components
## #
## # Edge Data: 20,304 x 4 (active)
##    from    to type        n
##   <int> <int> <chr>   <int>
## 1     1 14535 mention     1
## 2     2  1671 retweet     1
## 3     2 11379 retweet     1
## 4     3 14535 mention     1
## 5     4 14536 retweet     1
## 6     4  1364 retweet     1
## # ... with 2.03e+04 more rows
## #
## # Node Data: 17,384 x 6
##   name         community degree between     closeness    eigen
##   <chr>            <int>  <dbl>   <dbl>         <dbl>    <dbl>
## 1 ___Kiira____         1      1       0 0.00000000331 0.0154  
## 2 __Amalin__          23      2       0 0.00000000331 0.000491
## 3 __amoy_              1      1       0 0.00000000331 0.0154  
## # ... with 1.738e+04 more rows

4.4 Identify a prominent person in the network

Key players in the ego network and activity network have different interpretation. In this case, the key player will be considered as a person who makes this specific keyword viral. We also can identify which person whose opinions are widely agreed upon by others.

We got interesting results here. every top centrality has a different account. It means every top user is great in their own ‘job’ on the network. we can tell that there’s no super user in #NewNormal network. From the table above, “adrianoqalbi” grab my attention the most. he appears in 2 centrality (if we remove NA in eigenvector centrality). Thus we can conclude he is the key player in #NewNormal Twitter activity network. Let’s see his tweets

4.5 Visualize network

let’s try to visualize the network. the network consist of 23k+ nodes and 17k+ degree. we need to filter it otherwise it’ll be just hairball of nodes and edges. here i just plot top 3 community and filter the label by nodes with degree centrality >= 3 or betweenness centrality >= 100. The edges color are separated by interaction type.

What can we get from this visualization? First, lets see this table

Now we can identify which person belongs to which community based on highest degree. The plot is arranged by community and id (row number), so user in the same community are placed near to each other. community #1 (highest community by frequency) is the only community with 2 interaction types. lots of them are retweeting (or quoting) “adrianoqalbi” tweet, and the others are people who mention “ari_maj” in their tweets. Users in community #2 and #3 are just retweeting each other, or maybe some of them make a quote tweet that also goes viral. There’s a news account like (detikcom) in community #3, it’ll be our evidence that those communities are driven by retweet. I can conclude that the top 3 community (by highest frequency) in Twitter activity network is well separated by its tweet types (interaction types). maybe there’s another interesting insight in another community since it creates fewer accounts by each community.

5 Additional business case: Competition network

Competition network gathers all person/user who has connection to specific official account. The main idea is mapping the whole network of accounts in the same industry. for example, we want to see marketing competition in Indonesian private colleges. We gather all of their followers and from that follower, gather all their following. Build community and calculate the metrics, thus we will get important persons who has information from all competing accounts. we also can found which community is a potential ‘unreached’ market to several companies and how to reach them.

The graph above shows 21 Indonesian private college Instagram admission of new student network. The network is separated by 3 community, green for Muslim college, purple for Catholic/Christian college, and fuzzy cluster in the middle is what we can say as a ‘neutral’ college. We can conclude that these days people still looking for education based on their religion. Let’s say we’re a social media analyst from Binus, then we can find out how to reach UII/UMY/UMM cluster based on important person in their community (endorsement). It will give Binus a lot of engagement right?

Well, the problem is Twitter rate limitation makes it impossible to do this analysis in a short time. So i’ll tell you how to do it. It’s pretty simple, but takes a loong time:

  1. Select several accounts in the same industry
  2. Gather all of their followers
  3. From the follower, gather all of their following
  4. Filter to active user only (if possible)
  5. Build a network, create community and centrality
  6. Identify prominent user based on your case
  7. Visualize

Case Objectives:

  • Analyze Competition Network from (for example) @kfc_id, @mcd_id, @wendys, @phd account
    • Visualize the whole competition network
    • Identify fanbase cluster in each account
    • Calculate the metrics, and find out who is the key player in the whole network